script to generate report for user resource usage #199

asmacdo · 2024-09-25T20:07:00Z

This PR introduces 4 scripts, which do most of the heavy lifting to calculate EFS resource usage by each user. It is additionally a start on a general pattern by which we can perform other cron-type tasks.

The 4 scripts are currently used by hand, as documented in the README, though they are a start on what will eventually be fully automated.

This went through a few different patterns during development, including deployment using Kubernetes and relying on the Karpenter Autoscaler. If those patterns are needed again, Ive left a branch unsquashed for future usage. https://github.com/dandi/dandi-hub/tree/cron-experiments-unsquashed

.github/workflows/report.yaml

kabilar · 2024-11-07T00:05:29Z

Hi @asmacdo, just checking in to see how this report generation is going? Thanks.

asmacdo · 2024-11-11T18:51:24Z

Ran into some problems

The du script ran for about 50 minutes and then the pod disappeared without logs.

Worse it kicked my jupyterhub pod as well as another user.

[I 2024-11-11 17:57:57.999 JupyterHub log:192] 200 GET /hub/error/503?url=%2Fuser%2Fasmacdo%2Fterminals%2Fwebsocket%2F1 (@100.64.247.104) 7.52ms
[W 2024-11-11 17:57:59.266 JupyterHub base:1254] User asmacdo server stopped, with exit code: 1
[I 2024-11-11 17:57:59.266 JupyterHub proxy:357] Removing user asmacdo from proxy (/user/asmacdo/)

I think this means we need to take a different approach. By setting resource limits, we should have isolated our job from the other pods, but since I have no other logs about what happened here I think we need to take a more conservative approach that is completely isolated from user pods.

I did it this way because I thought it would be simpler, but if theres any chance that we affect a running user pod, we would be better off directly deploying a separate EC2 instance and bind the EFS directly, avoiding Kubernetes altogether.

kabilar · 2024-11-14T20:58:32Z

we would be better off directly deploying a separate EC2 instance and bind the EFS directly, avoiding Kubernetes altogether.

Thanks @asmacdo. That makes sense.

.github/scripts/cleanup-ec2.sh

.github/scripts/launch-ec2.sh

yarikoptic · 2024-12-12T18:07:32Z

.github/scripts/cleanup-ec2.sh

+set -e
+
+# Load environment variables from the file if they are not already set
+ENV_FILE=".ec2-session.env"


Suggested change

ENV_FILE=".ec2-session.env"

ENV_FILE="/run/user/$(id -u)/ec2-session.env"

yarikoptic

left comments on the report and walking.

The plan:

see if more recent AMI with newer than 3.7 python available (e.g. 3.10 to be future proof somewhat)
run the create-file-index.py across all the users
- could be done in parallel and under duct following http://www.oneukrainian.com/tmp/run_manifests.sh example
scp those reports somewhere
while running develop produce-report.py further:
- change logic so we compute sizes for all directories
- go through the list of folders and match to the desired to be monitored/informed about https://github.com/dandi/dandi-hub/pull/188/files#diff-b7bf13dc481c474b9080a061c3154f0e645619658cb9bd469383938de07a7439R16
  - ideally compose a dict/json record with those, do not render immediately. e.g.

{
   "total":
    {
      "file_size": 123123123123, 
      "file_count":123
    },
   "caches": {
        "__pycache__": {  # count directories which last path component is `__pycache__`
             "file_size": 12123123,
             "file_count": 12,
             "directories": [
                  "a/e/__pycache__"
                  "a/b/d/e/__pycache__"
                  "a/e/__pycache__/c/__pycache__"  # should not happen in the wild
             ],
        "pip_cache": {
               "file_size": 13,
               "file_count": 1,
               "directories": [
                    "blah/.cache/pip
               ]
   }

.github/scripts/create-file-index.py

yarikoptic · 2024-12-12T18:28:59Z

.github/scripts/create-file-index.py

+
+    files_metadata = []
+
+    for root, dirs, files in os.walk(directory):


FTR. walk seems to be already doing desired (right) thing and does not follow symlinked folders. We get

In [14]: list(os.walk('/tmp/1234')) Out[14]: [('/tmp/1234', ['infinitum', 'subdir', 'linkgood'], []), ('/tmp/1234/subdir', ['subdir2'], ['file']), ('/tmp/1234/subdir/subdir2', [], ['file2'])]

for

❯ tree /tmp/1234 /tmp/1234 ├── infinitum -> /tmp/1234 ├── linkgood -> subdir └── subdir ├── file └── subdir2 └── file2

note: we do not monitor empty folders below

.github/scripts/produce-report.py

asmacdo · 2024-12-12T23:01:26Z

see if more recent AMI with newer than 3.7 python available (e.g. 3.10 to be future proof somewhat)

Amazon Linux 2023 comes with 3.9, which is sufficient. I briefly tried ubuntu 22.04, (with 3.12) but that would add complexity, requiring amazon-efs-utils to be installed from source.

asmacdo · 2024-12-16T18:55:06Z

Update:

The automation is incomplete
- the launch script sets up an ec2 instance with the scripts
- scripts must be executed by hand
- produced data must be retrieved by hand
Running in parallel (8 simultaneous) the file indexing takes about 16 minutes
The file indexes are 499MB

Heres an example of the json output produced by the report generator:

{
  "total_size": 238294213,
  "file_count": 3639,
  "caches": {
    "pycache": {
      "total_size": 22782820,
      "file_count": 1431,
      "directories": [
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pkg_resources/extern/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pkg_resources/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pkg_resources/_vendor/jaraco/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pkg_resources/_vendor/jaraco/text/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pkg_resources/_vendor/importlib_resources/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pkg_resources/_vendor/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pkg_resources/_vendor/packaging/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pkg_resources/_vendor/pyparsing/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pkg_resources/_vendor/pyparsing/diagram/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pkg_resources/_vendor/more_itertools/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/_distutils/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/_distutils/command/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/extern/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/config/_validate_pyproject/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/config/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/_vendor/jaraco/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/_vendor/jaraco/text/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/_vendor/importlib_resources/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/_vendor/importlib_metadata/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/_vendor/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/_vendor/tomli/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/_vendor/packaging/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/_vendor/pyparsing/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/_vendor/pyparsing/diagram/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/_vendor/more_itertools/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/command/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/_distutils_hack/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/index/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/cli/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/req/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/operations/build/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/operations/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/operations/install/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/resolution/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/resolution/legacy/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/models/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/vcs/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/locations/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/network/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/metadata/importlib/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/metadata/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/distributions/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/webencodings/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/pkg_resources/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/idna/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/msgpack/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/certifi/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/colorama/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/tenacity/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/distro/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/caches/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/tomli/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/packaging/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/pyparsing/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/pyparsing/diagram/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/urllib3/util/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/urllib3/contrib/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/urllib3/contrib/_securetransport/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/urllib3/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/urllib3/packages/backports/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/urllib3/packages/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/distlib/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/chardet/cli/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/chardet/metadata/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/compat/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/pygments/styles/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/pygments/filters/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/pygments/lexers/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pkg_resources/extern/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pkg_resources/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pkg_resources/_vendor/jaraco/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pkg_resources/_vendor/jaraco/text/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pkg_resources/_vendor/importlib_resources/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pkg_resources/_vendor/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pkg_resources/_vendor/packaging/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pkg_resources/_vendor/pyparsing/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pkg_resources/_vendor/pyparsing/diagram/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pkg_resources/_vendor/more_itertools/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/_distutils/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/_distutils/command/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/extern/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/config/_validate_pyproject/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/config/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/_vendor/jaraco/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/_vendor/jaraco/text/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/_vendor/importlib_resources/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/_vendor/importlib_metadata/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/_vendor/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/_vendor/tomli/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/_vendor/packaging/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/_vendor/pyparsing/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/_vendor/pyparsing/diagram/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/_vendor/more_itertools/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/command/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/_distutils_hack/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/index/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/cli/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/req/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/operations/build/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/operations/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/operations/install/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/resolution/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/resolution/legacy/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/models/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/vcs/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/locations/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/network/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/metadata/importlib/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/metadata/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/distributions/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/webencodings/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/pkg_resources/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/idna/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/msgpack/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/certifi/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/colorama/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/tenacity/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/distro/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/caches/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/tomli/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/packaging/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/pyparsing/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/pyparsing/diagram/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/urllib3/util/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/urllib3/contrib/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/urllib3/contrib/_securetransport/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/urllib3/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/urllib3/packages/backports/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/urllib3/packages/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/distlib/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/chardet/cli/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/chardet/metadata/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/resolvelib/compat/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/resolvelib/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/platformdirs/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/pygments/styles/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/pygments/filters/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/pygments/lexers/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/con_duct/__pycache__",
        "asmacdo/dandi-notebooks/000971/lernerlab/seiler_2024/__pycache__"
      ]
    },
    "user_cache": {
      "total_size": 10852503,
      "file_count": 81,
      "directories": [
        "asmacdo/.cache"
      ]
    },
    "yarn_cache": {
      "total_size": 0,
      "file_count": 0,
      "directories": []
    },
    "pip_cache": {
      "total_size": 10852503,
      "file_count": 80,
      "directories": [
        "asmacdo/.cache/pip"
      ]
    },
    "nwb_cache": {
      "total_size": 0,
      "file_count": 0,
      "directories": []
    }
  }
}

Notably, the total size 238294213 is less than reported by du.
du --apparent: 247232941
du: 252841984

asmacdo · 2025-01-28T16:12:37Z

@kabilar Do you want to review this before I merge? Its not very clean, but IMO it would be better to just merge as-is and iterate on it next time we generate these reports.

README.md

Co-authored-by: Yaroslav Halchenko <[email protected]>

kabilar reviewed Sep 26, 2024

View reviewed changes

.github/workflows/report.yaml Outdated Show resolved Hide resolved

asmacdo force-pushed the cron-data-usage-report branch 3 times, most recently from a4b44f9 to 965a81e Compare November 11, 2024 16:55

asmacdo force-pushed the cron-data-usage-report branch from 81b43e0 to 747f0a4 Compare December 2, 2024 17:47

yarikoptic reviewed Dec 12, 2024

View reviewed changes

.github/scripts/cleanup-ec2.sh Outdated Show resolved Hide resolved

yarikoptic reviewed Dec 12, 2024

View reviewed changes

.github/scripts/launch-ec2.sh Outdated Show resolved Hide resolved

yarikoptic reviewed Dec 12, 2024

View reviewed changes

.github/scripts/launch-ec2.sh Show resolved Hide resolved

yarikoptic reviewed Dec 12, 2024

View reviewed changes

yarikoptic requested changes Dec 12, 2024

View reviewed changes

yarikoptic mentioned this pull request Dec 19, 2024

Delete files from backup that don't match any inventory items dandi/s3invsync#33

Closed

asmacdo force-pushed the cron-data-usage-report branch from fcd9531 to 845df00 Compare January 17, 2025 18:19

asmacdo mentioned this pull request Jan 21, 2025

(aspirational hack project) Auto-generate EFS usage report & Log storage #173

Closed

asmacdo marked this pull request as ready for review January 28, 2025 16:11

asmacdo changed the title ~~GH action to generate report~~ script to generate report for user resource usage Feb 4, 2025

yarikoptic reviewed Feb 4, 2025

View reviewed changes

README.md Outdated Show resolved Hide resolved

Add scripts to produce EFS usage report

cb8bb26

asmacdo force-pushed the cron-data-usage-report branch from b4cc442 to cb8bb26 Compare February 9, 2025 19:47

Update .github/scripts/cleanup-ec2.sh

57a07b2

Co-authored-by: Yaroslav Halchenko <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

script to generate report for user resource usage #199

script to generate report for user resource usage #199

asmacdo commented Sep 25, 2024 •

edited

Loading

kabilar commented Nov 7, 2024

asmacdo commented Nov 11, 2024 •

edited

Loading

kabilar commented Nov 14, 2024

yarikoptic Dec 12, 2024

yarikoptic left a comment

yarikoptic Dec 12, 2024

asmacdo commented Dec 12, 2024

asmacdo commented Dec 16, 2024

asmacdo commented Jan 28, 2025

	ENV_FILE=".ec2-session.env"
	ENV_FILE="/run/user/$(id -u)/ec2-session.env"


		files_metadata = []

		for root, dirs, files in os.walk(directory):

script to generate report for user resource usage #199

Are you sure you want to change the base?

script to generate report for user resource usage #199

Conversation

asmacdo commented Sep 25, 2024 • edited Loading

kabilar commented Nov 7, 2024

asmacdo commented Nov 11, 2024 • edited Loading

kabilar commented Nov 14, 2024

yarikoptic Dec 12, 2024

Choose a reason for hiding this comment

yarikoptic left a comment

Choose a reason for hiding this comment

yarikoptic Dec 12, 2024

Choose a reason for hiding this comment

asmacdo commented Dec 12, 2024

asmacdo commented Dec 16, 2024

asmacdo commented Jan 28, 2025

asmacdo commented Sep 25, 2024 •

edited

Loading

asmacdo commented Nov 11, 2024 •

edited

Loading